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ABSTRACT.  Clark  (1996)  has  identified  common  ground-information  we  take  to  be  shared  with  others-as  an  indispensable  requisite  for 
everything  people  do  with  each  other  where  language  use  or,  more  definitively,  a  collaborative  use  of  meaning  and  understanding  is  involved.  In 
interactive  software  development,  though,  theory  and  practice  have  placed  little  emphasis  on  the  role  and  importance  of  this  crucial  aspect  of 
cognition  in  user  interactions,  with  the  result  that  user  interface  and  interaction  designs  are  typically  impoverished  in  ways  they  need  not  be. 

In  this  paper,  the  basic  features  of  a  language  use  approach  to  human-computer  interaction  are  outlined,  and  a  range  of  both  noncomputational 
and  computational  implications  for  the  design  of  interactive  systems  is  examined.  In  particular,  human-computer  interaction  is  recast  as  a  genuine 
instance  of  language  use  between  the  user  and  the  system  designer  and,  in  a  second  layer,  as  a  joint  activity  in  which  the  system  and  the  user  are 
participants.  In  this  view,  principled  interaction  design  is  authorship  that  promotes  common  ground  with  the  user  at  all  times  in  all  layers  through 
noncomputational  and/or  computational  means.  Noncomputational  means  facilitate  the  user’s  development  and  maintenance  of  common  ground 
with  the  author’s  design  through  passive  and  semipassive  mechanisms  that  may  also  depend  on  user  initiative.  Computational  means  focus  on 
actively  maintaining  a  system-side  image  of  common  ground  and  using  this  to  inform  aspects  of  the  system’s  interaction  behavior.  Key  issues  in 
computing  common  ground  include  representing  and  verifying  what  the  user  does  and  does  not  know,  identifying  and  using  conventions,  and 
solving  coordination  problems  posed  by  the  user.  Ways  that  both  noncomputational  and  computationally  based  language  use  approaches  to 
human-computer  interaction  designs  can  better  support  user  comprehension  and  performance  through  the  promotion  and  use  of  common  ground, 
and  cognitive  issues  these  approaches  address,  are  discussed. 


1  Introduction 

Motivated  by  a  continuing  effort  to  appreciate  ways  in  which 
software  user  interfaces  can  be  better  made  to  support  users, 
this  short  paper  examines  at  a  high  level  how  and  why  the 
study  of  language  use  is  theoretically  relevant  to  the  design  of 
human-computer  interaction. 

At  its  heart,  the  study  of  language  use  is  concerned  with 
collaborations  where  one  person’s  meaning  and  another’s 
understanding  are  essential  to  carrying  out  what  they  are 
doing  together.  Meaning  and  understanding  are  conveyed 
through  the  use  of  signals,  and  any  device  that  can  serve  this 
purpose  well,  no  matter  if  it  is  a  gesture,  a  sign,  or  an 
utterance,  amounts  to  a  use  of  language.  (Think  of  the  adage 
"a  picture  is  worth  a  thousand  words.")  Arguably,  people’s 
language  use  skills — their  complementary  abilities  to  devise 
ways  to  indicate  their  intentions  and  to  successfully  resolve 
the  indications  of  others — are  their  most  fundamental 
resource  in  all  activities  that  require  collaboration. 

Observations  such  as  the  foregoing  about  language  use  have 
direct  implications  for  human-computer  interaction.  Although 
computers  are  far  from  the  equivalent  of  people,  people 
nevertheless  use  them  to  do  all  sorts  of  things  that  would  be 
difficult  at  best  to  do  on  one’s  own.  This  use  has  the  distinct 
character  of  a  collaboration.  People  interact  with  software 
through  user  interfaces  that  require  them  to  understand  their 
role  in  the  task  and  that  constrain  their  goals  and  demand  their 
cooperation  in  the  coordination  of  a  range  of  participatory 
actions. 

But  user  interfaces  are  not  devised  by  computers,  of  course. 
They  are  devised  by  people.  As  Terry  Windograd  noted  in  an 
interview  published  in  Preece,  et  al.  (1994),  "everything  that 
is  in  [a]  computer  came  from  somebody  in  some  context  with 
some  purpose  and  some  meaning."  The  implication  is  that  the 
design  of  human-computer  interaction  is  fundamentally  a 
representation  problem  that  involves  the  designer’s  meaning 
and  the  user’s  understanding. 


The  broader  implication  is  that  all  principles  of  language  use 
are  important  in  human-computer  interaction.  In  this  paper,  it 
is  argued  that  they  are  the  proper  basis  of  a  principled 
framework  for  interaction  designs. 

2  What  is  language  use? 

Herbert  Clark  (1996)  characterizes  the  study  of  language  use 
not  as  a  science  of  the  structure  of  language  but  instead  as  a 
cognitive  and  social  science  that  is  centrally  concerned  with 
the  notions  of  speaker’s  meaning  and  addressee’s 
understanding.  In  this  view,  all  instances  of  language  use  are 
instances  of  two  or  more  participants  acting  jointly  in  one 
setting  or  another,  though  not  necessarily  at  the  same  time, 
who  jointly  coordinate  individual  cognitive,  physical,  and 
perceptual  actions  as  needed  to  accomplish  a  social  purpose 
they  share.  The  setting  in  which  a  particular  instance  of 
language  use  occurs  determines  how  these  joint  actions  are 
coordinated  and  what  skills  are  needed.  Face-to-face  settings 
are  the  most  basic  and  written  settings  are  perhaps  the  most 
demanding.  Language  use  also  often  involves  more  than  one 
conceptual  domain  of  action.  These  domains  are  called  layers 
and  are  identified  by  references  to  persons  and/or  settings,  and 
possibly  other  matters,  that  are  not  necessarily  present  in  the 
initial  layer  of  action. 

Language  use  always  serves  people’s  larger  collaborative 
purposes — purposes  that  are  met  and  carried  out  socially  in 
their  joint  activities.  Every  instance  of  language  use,  though, 
is  itself  an  act  of  collaboration.  Signals  are  presented  with  the 
intention  of  conveying  meaning.  But  meaning  can  only  be 
understood  with  an  addressee’s  participation.  By  taking  up  a 
signal  and  grasping  its  sense  in  context,  addressees  make  the 
action  behind  the  signal’s  presentation  a  meaningful  and 
complete  joint  action.  The  principles  at  work  in  this  process 
are  the  primary  concern  of  language  use  studies. 

Thus,  meaning  and  understanding  are  central,  and  whenever 
these  notions  are  involved,  even  wholly  nonlinguistic 
collaborations  between  people  should  be  readily  identified  as 
instances  of  language  use.  Wholly  essential  for  meaning  and 
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understanding,  though,  is  the  notion  of  common  ground, 
which  Clark  identifies  as  an  indispensable  requisite  in  all 
things  people  undertake  to  do  with  each  other. 

2.1  What  is  common  ground? 

Common  ground  is  essentially  a  specific  kind  of  shared 
knowledge.  More  precisely,  it  is  knowledge  that  people  take 
to  be  held  in  common  on  the  basis  of  shared  information. 
People  try  to  justify  and  use  common  ground  by  presenting 
what  they  believe  is  an  adequate  shared  basis  for  it.  In 
general,  all  signals,  be  they  words,  images,  or  something  else, 
function  as  shared  information  when  they  are  presented  and 
perceived.  Suppose  that  two  people  have  agreed  to  go  to 
lunch  at  noon  and  one  shows  the  other  that  it  is  now  just  past 
twelve.  When  the  time  is  presented,  it  is  a  signal  that 
becomes  shared  information.  What  is  meant  and  understood 
by  it,  that  they  have  agreed  to  go  to  lunch  now,  is  knowledge 
that  resides  in  their  common  ground  on  the  basis  of  their 
earlier  agreement.  Presenting  the  time  both  justifies  and 
makes  use  of  this  common  ground  by  referring  to  their 
previous  agreement  concerning  lunch  at  noon  and  indicating 
that  it  is  time  to  go. 

Common  ground  is  built  up  moment-by-moment  over  a 
lifetime  of  joint  activities  and  experience  and  presupposes 
certain  cognitive  processes  (most  notably,  ordinary  perceptual 
skills,  the  workings  of  short  and  long  term  memory,  and 
mundane  reasoning  abilities).  Every  event  that  openly  takes 
place  and  is  mutually  attended  to  in  a  joint  activity 
immediately  becomes  a  piece  of  knowledge  in  its  participants’ 
common  ground.  Ordinarily,  this  includes  all  of  the  shared 
bases  people  use  with  each  other  and,  just  as  importantly, 
most  of  the  inferences  each  would  normally  make  in  context. 
Clark  argues  that  people  conceptually  divide  their  common 
ground  in  joint  activities  into  at  least  three  parts:  first,  the 
common  ground  they  bring  to  the  joint  activity;  second,  an 
idea  of  the  joint  activity’s  current  state;  and  third,  a  record  of 
what  has  been  done  so  far.  People  naturally  keep  track  of  this 
information  in  one  form  or  another,  but  just  as  importantly, 
they  expect  their  counterparts  to  do  very  much  the  same. 

2.2  Coordination  in  language  use 

Common  ground  makes  it  possible  for  people  to  cooperatively 
orient  each  other’s  cognition  with  signals.  To  direct  one 
person’s  understanding  to  what  another  means  by  a  particular 
signal  requires  each  individual  to  coordinate  a  corresponding 
set  of  interdependent  physical,  perceptual,  and  cognitive 
actions  with  his  or  her  counterpart.  This  coordinated  effort  is 
what  makes  each  instance  of  language  use  a  joint  action. 

Clark  characterizes  joint  actions  involving  meaning  and 
understanding  as  coordination  problems  that  people,  in  most 
instances  of  language  use,  pose  for  each  other  and  then 
immediately  work  together  to  solve.  To  be  efficient,  certain 
principles  apply.  Among  these  are  that  first,  a  straightforward 
solution  should  already  be  in  mind  when  the  problem  is 
posed.  Second,  all  of  the  information  needed  to  reach  the 
solution,  given  what  is  common  ground,  should  be  shared  in 
the  problem’s  posing  (this  information  functions  as  a 
coordination  device).  And  third,  this  solution  should  be  one 
that  is  readily  and  unambiguously  apparent  to  everyone  in 
context.  Respectively,  these  are  premises  of  solvability, 
sufficiency,  and  joint  salience.  Experience  motivates  people 


to  expect  these  premises  to  be  met  as  they  go  about 
coordinating  their  joint  actions;  otherwise,  a  larger  effort  is 
always  needed  to  accomplish  the  same  thing. 

Working  out  coordination  problems  is  a  dominant  part  of 
people’s  language  use  skills.  In  choosing  the  right  basis  to  use 
as  a  coordination  device,  joint  salience  is  usually  the  most 
important  consideration.  Capitalizing  on  this  aspect  of 
people’s  attention  in  context  greatly  increases  the  likelihood 
of  their  finding  the  intended  solution.  Two  important  sources 
of  coordination  devices  that  readily  meet  this  test  are  external 
representations  and  conventions.  External  representations  are 
simply  elements  of  the  physical  situation  in  a  setting  of 
language  use  that  can  be  used  to  represent  parts  of  the  joint 
activity  taking  place  there.  Not  all  external  representations 
have  the  same  properties  and  advantages,  but  they  generally 
make  superior  coordination  devices  because  of  their 
perceptual  immediacy  and  their  inherent  significance  in  the 
common  ground  of  the  moment.  Conventions  are 
representations  of  standing  solutions  to  frequently  occurring 
coordination  problems,  and  therein  lies  their  value:  they  are 
successful  and  familiar  regularities  of  action.  When  a 
convention  is  signaled  and  recognized  as  a  coordination 
device,  addressees  usually  know  instantly  how  to  proceed. 
Conventions  and  systems  of  conventions  are  preeminently 
useful  for  purposes  of  language  use.  Clark  notes  that, 
"Languages  like  English  are  conventional  signaling  systems 
par  excellence."  Conventions  and  external  representations  are 
often  found  together.  In  joint  activities  that  take  place  in 
written  settings,  for  instance,  the  media  itself  is  an  external 
representation  of  the  activity,  and  its  elements  serve  as  a 
continuous  stream  of  conventional  coordination  devices  (text, 
layout,  etc.)  that  addressees  use  to  complete  most  of  the  joint 
actions  the  writer  has  begun. 

3  Language  use  in  human-computer  interaction 

The  basic  claim  of  this  paper  is  that  human-computer 
interaction  is,  in  fact,  a  genuine  instance  of  language  use.  This 
is  the  case  largely  because  of  the  nature  of  its  supporting 
media:  software  user  interfaces  are  not  only  design  artifacts 
(i.e.,  products  of  human  design)  but  are  also  a  means  for 
carrying  out  a  class  of  joint  activities  (doing  computationally 
based  tasks)  in  which  all  principles  of  language  use  apply,  and 
whose  primary  participants  are  people  acting  as  themselves  in 
the  roles  of  designer  and  user.  In  contrast,  the  notion  of 
interacting  with  the  computer  (human-computer  interaction 
per  se)  is  simply  a  secondary  layer  of  action  on  top  of  this 
primary  one  (more  on  this  below). 

The  joint  activity  between  people  defined  by  human-computer 
interaction  is  made  up  of  joint  actions  that  arise  from  the 
coordination  of  individual  cognitive,  physical,  and  perceptual 
actions  on  the  part  of  both  the  designer  and  the  user,  albeit 
across  time.  These  joint  actions  implicitly  depend  on  the 
function  of  common  ground  and  always  involve  the 
designer’s  meaning  and  the  user’s  understanding.  Put  another 
way,  a  software  user  interface  presents  a  designer’s  notion  of 
how  a  particular  computational  task  can  be  organized  and 
carried  out.  When  a  user  takes  up  the  task,  part  of  what  he  or 
she  faces  is  the  job  of  solving  a  substantial  array  of 
coordination  problems  the  designer  has  posed.  Every  aspect 
of  the  presentation  is  available  to  be  used  as  a  coordination 
device,  and  each  solution  the  user  correctly  converges  on 
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makes  joint  a  corresponding  set  of  pre-coordinated  actions  the 
designer  has  initiated. 

3.1  Comparing  written  settings 

Because  the  designer’s  conception  has  been  worked  out  and 
produced  in  advance  of  its  presentation,  the  joint  activity  that 
takes  piace  in  the  primary  iayer  of  human-computer 
interaction  must  rightiy  be  characterized  as  one  that  takes 
piace  in  a  written  setting.  In  contrast  to  more  conventionai 
written  media  such  as  that  of  a  book,  the  scene  and  medium  of 
this  written  setting  are  those  of  a  computer’s  interactive 
dispiay.  Aii  written  presentations  share  certain  basic 
characteristics,  but  the  written  media  of  human-computer 
interaction  differs  from  conventionai  written  media  in  severai 
important  ways. 

3.1.1  Layers 

First,  uniike  the  second  iayer  of  action  that  is  ordinariiy 
presented  in  a  book  or  magazine,  wherein  the  reader  remains 
in  a  written  setting  but  participates  in  the  imagining  of  a  story 
or  the  domain  of  an  essay,  the  second  iayer  of  action  in 
human-computer  interaction,  perhaps  surprisingiy,  piaces  the 
user  in  a  face-to-face  setting.  Here,  the  user  is  expected  to 
interact  with  the  computer  to  carry  out  the  joint  activity 
impiicit  in  the  software  task,  as  if  the  computer  and  not  the 
designer  were  the  user’s  reai  counterpart.  This  shift  of  setting 
and  participants  between  iayers  naturaiiy  makes 
correspondingiy  different  demands  on  the  user’s  ianguage  use 
skiiis  and  aiso  has  potentiaiiy  damaging  consequences  for  the 
function  of  common  ground. 

3.1.2  Presentation  properties 

A  second  key  difference  between  conventionai  written  media 
and  that  of  human-computer  interaction  can  be  found  in  the 
organization  and  expectations  of  the  iatter’s  presentation, 
which  is  noniinear  and  deiiberateiy  opportunistic.  This  is 
decidediy  different  from  the  iinear  presentation  that  is  the 
encouraged  convention  in  most  other  written  settings.  Instead, 
in  human-computer  interaction,  the  user  has  opportunistic 
controi  of  the  joint  activity.  Inevitabiy,  this  ieads  to  a 
contingent  branching  of  focus  that  no  nontriviai  design  can 
fuiiy  anticipate  for  the  user.  Hence,  when  designs  reiy  on 
dependent  concepts  for  the  soiution  of  wideiy  divergent  goais, 
as  they  sometimes  must,  users  understandabiy  may  not  know 
what  to  do  next  through  no  fauit  of  their  own.  Users  often 
have  iittie  recourse,  though,  but  to  depend  on  the  designer,  in 
posing  coordination  probiems,  to  honor  their  naturai 
expectations  of  soivabiiity,  sufficiency,  and  joint  saiience. 

Severai  other  presentation  properties  are  worth  highiighting 
at  this  point.  Human-computer  interaction  reiies  on  a 
ianguage  of  coordination  devices  that  is  made  up  of  a  variety 
of  iinguistic  and  noniinguistic  signais  inciuding  eiements  of 
naturai  ianguage,  visuai  artifacts,  and  behaviors  (i.e.,  actions 
and  procedures).  Many  of  these  presentation  eiements,  in  the 
modem  cuiture  of  computer  iiteracy,  are  rightiy  intended  to  be 
constmed  as  conventions.  The  presentation  aiso  serves  in  part 
as  an  externai  representation  of  the  joint  activity.  This  aspect 
of  a  software  user  interface  is  usuaiiy  intended  to  function  as 
a  manipuiabie  modei  of  the  task  at  hand,  but  does  not  aiways 
fuiiy  indicate  the  task’s  status.  For  instance,  actions  that  take 
piace  in  the  externai  representation  are  often  evanescent  to  the 
extent  that  a  user  may  have  to  return  to  an  eariier  procedure  to 


confirm  the  outcome  of  a  previous  event.  And  whiie  the 
externai  representation  is  accessibie  to  both  the  user  and  the 
system,  it  is  mostiy  unused  by  the  system  as  a  means  for 
justifying  common  ground  in  the  second  iayer  of  action. 

3.1.3  Common  ground 

Common  ground  turns  out  to  be  buiit  differentiy  in  each  of 
the  first  two  iayers  of  human-computer  interaction,  and  this 
proves  to  be  a  third  important  way  user  interfaces  understood 
as  written  media  differ  in  their  use  from  conventionai  written 
presentations.  For  both  the  designer  and  the  user,  finding 
common  ground  is  an  intuitive  prerequisite  for  the  conduct  of 
their  joint  activity.  More  often  than  not,  a  presentation’s  use 
of  various  sorts  of  dispiay  conventions  wiii  be  enough  to  get 
this  process  started.  Unfortunateiy,  though,  because  software 
user  interfaces  are  conventionaiiy  organized  as  noniinear 
presentations,  the  designer’s  controi  over  the  process  of 
buiiding  a  compiete  body  of  common  ground  with  the  user  in 
the  pianned  manner  of  a  iinear  presentation  is  sacrificed. 
Lineariy  organized  presentations  in  conventionai  written 
media  have  an  important  strength:  they  naturaiiy  correspond 
to  what  Ciark  caiis  the  third  part  of  common  ground  in  joint 
activities,  i.e.,  a  record  of  what  has  been  done  so  far.  This 
correspondence  permits  addressees  to  use  such  presentations 
to  their  advantage;  they  are,  for  instance,  intuitiveiy  indexed 
and  readiiy  open  to  review  when  misunderstandings  or 
questions  arise.  However,  in  the  written  setting  of  human- 
computer  interaction  (specificaiiy,  in  its  first  iayer),  the 
designer  can  oniy  buiid  common  ground  with  the  user  in  an 
irreguiar  manner.  The  reason  for  this  iimitation  is  the 
opportunistic  nature  of  the  presentation.  That  is,  opportunities 
for  the  designer  to  informaiiy  promote  aii  of  the  common 
ground  the  user  wiii  need  to  efficientiy  coordinate  any  part  of 
the  joint  activity  pubiished  in  the  design  are  effectiveiy 
contingent  on  what  the  user  chooses  to  do.  For  users,  a  further 
difficuity  with  written  presentations  of  this  sort  of  lies  in  their 
inherent  iack  of  correspondence  with  the  third  part  of 
common  ground  and  the  advantages  that  wouid  attend.  As  a 
consequence,  over  repeated  episodes  of  use,  software  users 
often  deveiop  a  paipabie  sense  of  where  their  common  ground 
with  a  design  is  missing  and  wiii  even  iook  for  it  on  occasion 
uniess  the  effort  proves  to  be  too  costiy. 

Buiiding  common  ground  with  the  designer  in  a  user 
interfaces’s  first  iayer  of  action  is,  of  course,  whoiiy  reievant 
to  working  directiy  with  the  operation  of  the  computer  in  the 
second  iayer.  By  design,  second  iayer  interactions  are  in  many 
respects  intended  to  resembie  the  reguiar  conduct  of  joint 
activities  in  face-to-face  settings,  aibeit  with  a  sophisticated 
information  processing  machine.  Ordinariiy  in  face-to-face 
interactions,  peopie  expect  each  other  to  keep  track  of  what 
they  are  doing  together  as  part  of  a  transparent  exercise  of 
their  mundane  ianguage  use  skiiis.  Indeed,  people  nominally 
do  this  in  any  activity  they  undertake,  even  if  others  are  not 
involved.  Accordingly,  it  can  readily  be  argued  that  the 
knowledge  people  accumulate  and  are  able  to  justify  in  all  of 
their  undertakings  corresponds  structurally  to  Clark’s  three 
parts  of  common  ground:  knowledge  initially  held  about  an 
activity  at  its  start,  knowledge  of  the  activity’s  current  state, 
and  knowledge  of  what  has  taken  place  so  far. 

When  this  kind  of  knowledge  is  acquired  in  an  interactive 
setting  with  other  people,  it  functions  as  common  ground 
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whenever  shared  informational  bases  exist  to  justify  it  as 
such.  When  it  is  acquired  in  another  sort  of  interactive  setting, 
one  where  the  activity  involves  only  components  of  the 
environment  that  maintain  no  such  knowledge,  it  can  only  be 
construed  pragmatically  as  an  individual’s  lone  conception  of 
the  activity.  The  knowledge  users  acquire  in  the  second  layer 
of  human-computer  interaction,  though,  falls  somewhere  in 
between  these  two  extremes — for  the  most  part,  it  is  only  the 
user’s  conception  of  the  activity,  but  in  certain  ways,  it  is  also 
part  common  ground.  Put  more  explicitly,  in  face-to-face 
interactions  with  computers,  users  regularly  keep  track  of 
contingent  interaction  knowledge  corresponding  to  the  three 
parts  of  common  ground  as  well  as  any  informational  bases 
that  may  be  needed  to  justify  it.  Throughout  the  course  of  the 
interaction,  the  software  keeps  track  of  some  of  this  same 
knowledge  and  shares  corresponding  informational  bases  in 
limited,  ad  hoc  ways  (with  undo  mechanisms,  for  instance, 
and  other  kinds  of  interaction  histories).  Contingent 
interaction  knowledge  the  software  fails  to  be  able  to  justify 
and  use,  though,  cannot  be  construed  as  common  ground  in 
the  setting  nor  the  layer.  Failures  of  this  sort  have  the  effect  of 
withering  most  expectations  of  the  computer’s  capacity  to 
keep  track  of  the  joint  activity  as  another  person  would.  As  a 
consequence,  many  of  the  user’s  face-to-face  language  use 
skills  go  largely  unused. 

3.2  Design  challenges 

As  the  foregoing  discussion  has  attempted  to  illustrate,  when 
the  normal,  linearly  accumulative  function  of  common  ground 
in  a  language  use  setting  is  impaired,  people  are  naturally 
forced  to  work  harder  cognitively  to  accomplish  their  goals. 
Unfortunately,  this  is  the  case  in  the  settings  of  both  layers  of 
human-computer  interaction.  Since  opportunistic  interaction 
inherently  builds  incomplete  common  ground  in  the  first  layer 
of  an  interaction  design,  straightforward  design  strategies  that 
reward  user  initiative  with,  for  instance,  inexpensive,  well 
indexed,  at  hand  access  to  missing  knowledge  when 
discrepancies  are  encountered  and/or  simple,  unobtrusive 
ways  to  document  procedures  in  context  are  certainly  worth 
considering.  A  more  difficult  challenge  obtains  in  the  design 
of  second  layer  interactions.  Flere,  the  clear  need  is  to  devise 
further  reliable  ways  for  user  interfaces  to  accumulate,  justify, 
and  use  additional  contingent  interaction  knowledge  as  it 
develops  in  a  given  instance  of  human-computer  interaction. 
With  well  designed  access  to  second  layer  common  ground, 
users  should  be  able  to  make  greater  and  more  efficient  use  of 
their  face-to-face  language  use  skills  and  the  cognitive 
resources  these  skills  support. 

4  Research  approaches 

As  an  adjunct  to  the  short  review  of  basic  design  challenges 
for  language  use  in  human-computer  interaction  presented  at 
the  end  of  the  previous  section,  the  remainder  of  this  paper 
briefly  describes  two  fundamental  approaches  for  design 
research  in  this  area,  one  targeted  principally  at  issues  of 
usability  and  the  other  at  the  problem  of  face-to-face 
interactions  with  user  interfaces 


4.1  The  noncomputational  approach 

The  essential  thrust  of  this  approach  is  to  revisit  the  notion  of 
user-centered  design  (Norman  and  Draper,  1986)  through  the 
lens  of  language  use  principles.  Common  ground  in  this  view 
is  seen  as  a  sine  qua  non  for  usability.  Strategies  worth 
investigating  under  this  heading  include  design  reviews  that 
evaluate  both  the  presentation  language  as  a  set  of  shared 
informational  bases  and  the  interaction  design  as  a  set  of 
coordination  problems.  Such  reviews  should  focus  on  issues 
of  intended  meaning  and  the  user’s  expectations  of 
solvability,  sufficiency,  and  joint  salience.  Certain  legacy 
strategies  that  deserve  renewed  investigation  as  user  initiated 
means  for  addressing  the  inherent  first  layer  difficulty  of 
building  adequate  common  ground  also  fall  into  this 
approach.  These  strategies  include  the  design  of  robust,  low 
cost,  point-of-use  reference,  tutorial,  annotation,  and 
interaction  history  mechanisms. 

4.2  The  computational  approach 

The  long  range  goal  of  this  approach  is  to  share  the  user's 
cognitive  load  and  promote  the  use  of  face-to-face  language 
use  skills  in  the  second  layer  of  interaction  designs.  Although 
many  behaviors  exhibited  by  user  interfaces  can  be  easily 
misconstrued  as  evidence  of  regular  maintenance  of  common 
ground  by  a  computer  (e.g.,  various  reflexive  mechanisms, 
corrective  and  alerting  behaviors,  and  history  mechanisms  in 
general),  fully  functional  common  ground  between  users  and 
computers  can  only  arise  when  the  computer  maintains  a 
system-side  representation  of  the  three  parts  of  common 
ground  in  user  interactions  and  can  justify  and  use  this 
knowledge  on  the  basis  of  shared  information  to  advance  the 
joint  activity  in  reliable  and  productive  ways.  Ultimately,  this 
is  a  design  problem  for  artificial  intelligence  techniques 
(Brock  and  Trafton,  1999).  Most  aspects  of  reasoning  about 
common  ground  are  computationally  challenging.  Significant 
computational  goals  include  representing  and  verifying  what 
the  user  does  and  does  not  know  about  the  joint  activity, 
perceiving  and  solving  coordination  problems  posed  by  user, 
and  identifying  and  using  conventions  (Alterman  and 
Garland,  2001).  Design  issues  raised  by  these  goals  include 
matters  of  system  initiative,  appropriate  knowledge 
boundaries  (reasoning  about  content  vs.  reasoning  about 
functionality)  and  the  design  of  face-to-face  interactions  about 
interactions,  which  are  necessary  to  more  fully  support  users' 
language  use  skills  in  second  layer  transactions. 
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Motivating  question 


How  can  user  interfaces  better  support  users? 
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Observations 


•  People  use  language  (in  all  senses  where 
meaning  and  understanding  are  needed)  to 
do  things  with  each  other,  e.g.,  to 
collaborate 


•  People's  language  use  skills  are  their  most 
fundamental  resource  in  all  interaction 
contexts  involving  meaning  and 
understanding 

•  Although  people  interact  with  computers  to 
do  things,  people  are  also  the  authors  of 
human-computer  interaction  designs 
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Implications 


•  The  design  of  human-computer  interaction  is 
a  representation  problem  that  involves  the 
designer's  meaning  and  the  user's 
understanding 

•  Accommodating  the  principles  of  how  people 
use  language  is  relevant  as  a  framework  for 
the  design  of  human-computer  interaction 
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What  is  language  use? 


Language  use 

-  allows  people  to  carry  out  joint  activities 

-  is  a  collaborative  form  of  joint  action  built  on 
individual  actions 

-  always  involves  at  least  one  person's  meaning  and 
another's  understanding 

-  depends  on  representation,  i.e,  the  use  of  signs  and 
symbols  as  signals 

-  is  a  cognitive  science  and  a  social  science 

-  is  not  the  study  of  linguistics 
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Principles  of  language  use 


•  Language  is  used  to  carry  out  joint  activities 


•  Language  use  is  defined  by  joint  actions  that  arise  from 
the  coordination  of  cognitive,  physical,  and  perceptual 
actions  people  (e.g.,  speaker  and  addressee)  carry  out 
as  individuals 


•  A  joint  activity's  setting  determines  how  these  joint 
actions  are  coordinated  and  what  skills  are  needed 

•  Face-to-face  settings  are  the  most  basic,  written 
settings  are  the  most  demanding 

•  Meaning  and  understanding  in  joint  actions  requires 
common  ground 
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What  is  common  ground? 


Common  ground 


-  is  knowledge  people  take  to  be  held  in  common  on 
the  basis  of  shared  information 


•  People  try  to  justify  and  use  common  ground  by 
presenting  what  they  believe  are  shared  bases  for  it 

•  Signals  in  use  among  people  (e.g.,  words  and  gestures) 
are  examples  of  such  shared  bases  of  information 

•  Common  ground  presupposes  certain  cognitive 
processes  and  is  built  up  moment-by-moment  over  a 
lifetime  of  joint  activities  and  experience 

Three  parts  of  common  ground  in  joint  activities: 

-  knowledge  initially  taken  to  be  held  in  common 

-  knowledge  of  the  joint  activity's  current  state 

-  knowledge  of  what  has  openly  taken  place  so  far 


Derek  Brock,  Navy  Center  for  Applied  Research  in  Artificial  Intelligence 


7 


More  principles  of  language  use 


•  People  expect  each  other  to  accrue  and  use  knowledge 
and  information  corresponding  to  common  ground 


•  Each  joint  action  involving  meaning  and  understanding 
is  a  coordination  problem  people  must  try  to  solve 
together 


•  When  people  pose  coordination  problems  to  each  other, 
they  usually  have  specific  solutions  in  mind  and  can  be 
expected  to  present  the  most  salient  and  sufficient 
bases  with  respect  to  their  common  ground  as 
coordination  devices  for  indicating  these  solutions 


•  External  representations  and  conventions  are  excellent 
sources  of  coordination  devices 


•  Language  use  frequently  involves  more  than  one 
conceptual  layer  of  activity 
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Language  use  in  human-computer 
_ interaction _ 

Claim:  human-computer  interaction  is  a  genuine  instance 
ianguage  use 


-  It  is  a  layered  joint  activity  (doing  a  computational  task)  in 
which  all  principles  of  language  use  apply,  whose  primary 
participants  are  the  designer  and  the  user  in  these  roles 


-  The  joint  actions  that  make  up  this  joint  activity  arise  from 
the  coordination  of  individual  cognitive,  physical,  and 
perceptual  actions 


-  These  joint  actions  depend  on  common  ground  and  always 
involve  the  designer's  meaning  and  the  user's 
understanding 

-  It  takes  place  in  a  particular  type  of  written  setting  whose 
scene  and  medium  are  a  pre-planned,  interactive 
computational  presentation 
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Comparing  written  settings 


Human-computer  interaction  differs  from  a  conventional  written  setting: 

•  Second  layer  of  action  takes  place  in  a  face-to-face  setting 

•  Presentation  is  nonlinear  and  deliberately  opportunistic 

•  Common  ground  is  built  differently  in  both  first  and  second  layers 

•  External  representation  of  joint  activity  is  dynamic,  partly  evanescent 


Book 

Human-computer  interaction 

Joint  activity 

Setting 

Presentation 

Participants 

First  layer 

Second  layer 

Common  ground  -  initial 
Common  ground  -  state 
Common  ground  -  events 
Meaning  &  understanding 
Coordination  devices 
External  representation 
Conventions 

Relating  subject  matter  of  book 

Written  in  all  layers 

Linear,  static 

Writer,  reader;  pos.  others  in  other  layers 
Writer,  reader  as  themselves 

Writer,  reader  immersed  in  subject  matter 
Conventional  literacy,  introduction 
Corresponds  to  bookmark 

Corresponds  to  previous  portion  of  book 
Writer  poses  coordination  problems 
Elements  on  page 

Printed  matter 

Written  language,  organization 

Relating  and  doing  computational  task 
Written  in  L‘  layer,  face-to-face  in 
Nonlinear,  opportuni  Stic 

Designer,  user;  computer  in  layer 

Designer,  user  as  themselves 

User  and  computer  carrying  out  task 
Computer  literacy,  tutorial 

May  correspond  to  state  of  display 
Corresponds  to  interaction  history 
Designer  poses  coordination  problems 
Elements  and  behavior  of  display 
Dynamic,  partly  evanescent  display 
Actions,  display  elements,  input  devices 
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Layers  in  human-computer  interaction 


Human-computer  interaction  has  two  principal  layers  of 
activity 

First  layer: 

•  Designer  and  user  carry  out  a  joint  activity  whose 
purpose  is  to  carry  out  a  computational  task  as 
themselves  in  these  roles 

•  Takes  place  in  a  written  setting 
Second  layer: 

•  User  interacts  with  the  computer  to  carry  out  the  same 
joint  activity  as  if  the  computer  were  the  user's 
counterpart 

•  Takes  place  in  a  face-to-face  setting 

•  Each  layer  makes  different  demands  of  the  user's 
language  use  skills 
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Presentation  properties  of  human- 
computer  interaction 


The  presentation: 

•  serves  as  an  external  representation  of  the  joint  activity 

•  poses  a  fixed  set  of  coordination  problems  (joint  actions) 
whose  solution  depends  on  the  user's  participation  (i.e., 
comprehension  and  physical  interaction) 

•  uses  a  language  of  coordination  devices  made  up  of  a 
variety  of  linguistic  and  nonlinguistic  signals,  including 
elements  of  natural  language,  visual  artifacts,  and 
behaviors  (i.e.,  actions  and  procedures) 

•  depends  on  many  elements  of  the  presentation  language 
to  be  construed  as  conventions 


Note:  users  depend  on  the  designer,  in  posing  coordination 
problems  for  carrying  out  the  joint  activity,  to  honor  their 
expectations  of  salience,  sufficiency,  and  solvability 
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Additional  presentation  properties  of 
human-computer  interaction 

Presentation  is  nonlinear  and  opportunistic: 

•  User  has  opportunistic  control  of  joint  activity 


•  Opportunistic  design  gives  direct,  nonlinear  access  to 
starting  points  for  carrying  out  goals  and  subgoals 

•  Elements  of  the  joint  activity  anticipated  by  the  design  are 
presented  as  planned,  linear  interaction  sequences 

•  Branching  afforded  by  design  inherently  leads  to  contingent 
paths  among  planned  interaction  sequences 

Presentation's  external  representation  of  the  joint  activity: 

•  Serves  in  part  as  a  manipulable  model  of  the  joint  activity, 
but  does  not  always  indicate  its  state 


•  Is  accessible  to  both  the  user  and  the  system  but  is  mostly 
unused  by  the  system  to  justify  common  ground 

•  Actions  in  the  external  representation  are  often  evanescent 
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Knowledge  people  accumulate  about 

interactions 


Claim:  In  any  interactive  activity,  the  knowledge  people 
accumulate  and  are  able  to  justify  corresponds 
structurally  to  the  three  parts  of  common  ground: 

-  knowledge  initially  held  about  the  activity 

-  knowledge  of  the  activity's  current  state 

-  Knowledge  of  what  has  taken  place  in  the  activity 


Knowledge  of  an  activity: 

•  is  common  ground  when  the  interaction  takes  place 
between  people  and  shared  informational  bases  exist  to 
justify  it  as  common  ground 

•  is  only  an  individual's  conception  of  the  activity  when 
the  interaction  takes  place  with  elements  of  the 
environment  that  accrue  no  such  knowledge 

•  is  part  common  ground  and  part  individual  conception  in 
human-computer  interaction 
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Common  ground  in  the  first  layer  of 
human-computer  interaction 

In  a  nonlinear  presentation: 


•  Both  the  designer  and  the  user  strive  to  find  and  build 
common  ground,  but... 


•  The  designer's  control  over  the  process  of  building  a 
complete  body  of  common  ground  with  the  user  in  the 
planned  manner  of  a  linear  presentation  is  sacrificed 


•  The  user's  common  ground  with  the  designer  builds 
contingently  on  the  basis  of 

-  Opportunistically  becoming  familiar  with  the  presentation 
language 

-  Opportunistically  solving  coordination  problems  posed  in  the 
interaction  design 


•  Over  time,  the  user  develops  a  sense  of  where  common 
ground  is  missing  and  will  often  look  for  it  unless  the 
effort  proves  to  be  too  expensive 
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Common  ground  in  the  second  layer  of 
_ human-computer  interaction _ 

When  the  user  engages  the  computer  through  a  software 
user  interface: 


•  The  interaction  is  designed  to  resemble  a  joint  activity  in 
a  face-to-face  setting 

•  The  user  actively  keeps  track  of  contingent  interaction 
knowledge  corresponding  to  three  parts  of  common 
ground  as  well  as  informational  bases  to  justify  it 

•  At  most,  software  keeps  track  of  some  of  the  same 
knowledge  and  shares  informational  bases  in  limited,  ad 
hoc  ways,  such  as  undo  mechanisms  and  other  kinds  of 
interaction  histories 

•  Contingent  interaction  knowledge  the  software  fails  to  be 
able  to  justify  and  use  cannot  be  construed  as  common 
ground  in  the  setting  nor  the  layer 

•  User's  face-to-face  language  use  skills  go  largely  unused 


Derek  Brock,  Navy  Center  for  Applied  Research  in  Artificial  Intelligence 


16 


Improving  language  use  in  human- 
computer  interaction 


Claim:  when  the  normal  function  of  common  ground  in  a 
language  use  setting  is  impaired,  people  are  forced  to 
work  harder  cognitively  to  accomplish  their  goals 


Since  opportunistic  interaction  inherently  builds  incomplete 
common  ground  in  the  first  layer  of  an  interaction  design, 

-  users  need  inexpensive  access  to  missing  knowledge  when 
they  encounter  discrepancies  in  their  common  ground 

-  users  need  inexpensive  ways  to  off  load  and/or  confirm  their 
understanding 

Reliable  use  of  additional  contingent  interaction  knowledge 
by  the  user  interface  in  the  second  layer  of  human- 
computer  interaction  can  give  users  opportunities  to: 

-  make  increased  use  of  their  face-to-face  language  use  skills 

-  make  more  efficient  use  of  their  cognitive  resources 


Derek  Brock,  Navy  Center  for  Applied  Research  in  Artificial  Intelligence 


17 


Complementary  approaches  to  language 
use  in  human-computer  interaction 

Noncomputational  approach 


•  Common  ground  between  user  and  designer  in  their  roles 
as  participants  in  the  joint  activity  of  doing  a  computational 
task  is  a  sine  qua  non  for  usability 

•  This  is  essentially  user-centered  design  viewed  from  a 
language  use  perspective 

•  Empowers  user;  applies  mostly  to  first  layer  concerns 


Computational  approach 

•  Many,  if  not  most,  aspects  of  reasoning  about  common 
ground  are  computationally  challenging 

•  Ultimately,  this  is  a  design  problem  for  artificial  intelligence 
techniques 

•  Potential  for  sharing  user's  cognitive  load  and  capitalizing 
on  user's  language  use  skills  in  revisions  and  repairs 
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Noncomputational  approach 


Goal:  to  improve  usability  in  fixed  interaction  designs  through 
the  application  of  principles  of  language  use 


As  part  of  the  usability  engineering  process, 

•  evaluate  the  presentation  language  as  a  set  of  shared 
informational  bases  for  knowledge  about  the  design  that 
should  be  taken  as  common  ground  for  the  joint  activity: 

-  Look  for  shared  bases  that  are  ambiguous,  obscure,  or  indicate 
knowiedge  that  shouid  not  be  common  ground 

-  Evaiuate  use  of  convention  and  noveity 

•  evaluate  the  interaction  design  as  a  set  of  coordination 
problems  and  look  for  violations  of  language  use  principles: 

-  Are  expectations  of  saiience,  sufficiency,  and  soivabiiity  met? 

-  Do  interaction  resuits  confirm  user's  intentions? 

-  Can  misunderstandings  arise  due  to  timing  or  coordination? 
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More  on  Noncomputational  Approach 


Provide  enhanced,  interactively  inexpensive  ways  for  users  to 
find  and  reinforce  common  ground  on  their  own  such  as: 

•  point  of  use  reference  and  annotation  mechanisms  that 
take  user  to  relevant  entries  in  glossaries,  procedural 
descriptions  and  limitations,  tutorials,  personal  notes,  etc. 

-  Dowdy  but  fertile  area  for  additional  interaction  design 
research 


-  Minimize  cost  of  interruption  in  terms  of  leaving,  using,  and 
returning  to  reference,  and  returning  to  point  of  use 

•  full  interaction  history  mechanisms  that  can  be  used  and 
reviewed 


-  Interaction  histories  correspond  to  third  part  of  common 
ground  in  joint  activities 

-  Interaction  design  and  use  issues  similar  to  those  above  apply 
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Computational  approach 


Goal:  to  share  user's  cognitive  load  and  promote  user's 

language  use  skills  in  the  face-to-face  setting  (second  layer) 
of  human-computer  interaction 


Note  that  many  behaviors  in  user  interface  can  be  confused 
with  maintenance  of  common  ground  by  computer 

-  Reactive  behaviors  (e.g.,  passive  interactions  with  cursor) 

-  Alerting  and  corrective  behaviors  (e.g.,  spelling,  grammar) 

-  History  mechanisms  (e.g.,  undo,  recent  documents,  etc.) 

Functional  common  ground  between  the  user  and  the  computer 
can  only  arise  when  the  computer: 

•  maintains  a  system-side  representation  of  the  three  parts  of 
common  ground  in  user  interactions 

•  can  justify  this  knowledge  on  the  basis  of  shared  information 

•  can  use  this  knowledge  to  advance  the  joint  activity 
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More  on  computational  approach 


Challenges  and  issues... 

•  Significant  computational  goals: 

-  Representing  and  verifying  what  the  user  does  and 
does  not  know  about  the  joint  activity 

-  Perceiving  and  solving  coordination  problems  posed 
by  user 

-  Identifying  and  using  conventions 

•  Issues: 

-  Designing  face-to-face  interactions  about  interactions  to 
capitalize  on  user's  language  use  skills 

-  What  knowledge  is  appropriate  for  common  ground? 
(reasoning  about  content  vs.  reasoning  about  functionality) 

-  Shared  knowledge  the  system  can  justify  can  also  be  used 
as  a  basis  for  initiative  -  when  is  initiative  appropriate? 
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Relevant  work  at  NRL 


•  Collaborative  Engineering  Environments  (ONR) 

-  Design  studies  (with  H.  Ng,  NRL) 

-  WARCON  CEE  IPX  member  (ONR) 

-  Language  Use  Principles  for  Developing  Collaborative 
Environments  (Brock,  2000,  unpublished  paper) 


•  Task  model  tracing  (NRL,  ONR) 

-  ACT-R  model  of  resource  allocation  task  used  to  trace  user 
interactions  and  compute  common  ground 

-  A  Language  Use  Approach  to  Human-Computer  Interaction 
(Brock,  2001) 

•  Alerting  and  Interruption  (ONR) 

-  Multimodal  interface  studies  (with  J.  Balias,  NRL) 

-  Interruption  studies  (with  G.  Trafton,  NRL) 
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